Image segmentation based on block averaging

ABSTRACT

A method and system for improving the quality of a video image ( 100 ) segmented into a plurality of blocks ( 110, 115, 120 ) of known size is disclosed. The method comprises the steps of associating a value to each of said blocks and altering said associated value corresponding to a selected one of said blocks when each of said associated values of blocks adjacent to said selected block is different than said selected block associated value. The block value is a first value when said block probability function is greater than a threshold value, otherwise it a set as a second value.

BACKGROUND OF THE INVENTION

This invention relates to video processing and more specifically toclassifying and segmenting regions of pixels base upon characteristicssuch as color and texture.

SUMMARY OF THE INVENTION

A method and system for improving the quality of a video image segmentedinto a plurality of blocks of known size is disclosed. The methodcomprises the steps of associating a value to each of said blocks andaltering said associated value corresponding to a selected one of saidblocks when each of said associated values of blocks adjacent to saidselected block is different than said selected block associated value.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 illustrates a segment of an image organized in 8×8 pixel blocks;

FIG. 2 illustrates a flow chart an exemplary process for an improvedsegmentation method in accordance with the principles of the invention;

FIG. 3 illustrates a flow chart an exemplary second process for animproved segmentation method in accordance with the principles of theinvention;

FIG. 4 illustrates a system for executing the processing shown in FIGS.2 and 3.

It is to be understood that these drawings are solely for purposes ofillustrating the concepts of the invention and are not intended as adefinition of the limits of the invention. The embodiments shown inFIGS. 1 through 4 and described in the accompanying detailed descriptionare to be used as illustrative embodiments and should not be construedas the only manner of practicing the invention. The same referencenumerals, possibly supplemented with reference characters whereappropriate, have been used to identify similar elements.

DESCRIPTION OF THE INVENTION

Segmentation of video images, such as television images, is the processwherein each frame of a sequence of images is subdivided into regions orsegments. Each segment includes a cluster of pixels that encompass aregion of the image with common properties or characteristics. Forexample, a segment may be distinguished by a common color, texture,shape, amplitude range or temporal variation. Several methods are knownfor image segmentation using a process wherein a binary decisiondetermines how the pixels will be segmented. According to such aprocess, all pixels in a region either satisfy a common criteria for asegment and are therefore included in the segment, or they do notsatisfy the criteria and are completely excluded: While thesesegmentation methods are satisfactory for some purposes, they areunacceptable for many others. In the case of moving image sequences,small changes in appearance, lighting or perspective may only causesmall changes in the overall appearance of the image. However,application of a segmentation method such as that described above tendsto allow regions of the image that should appear to be the same tosatisfy the segmentation criteria in one frame, while failing to satisfyit in another. One of the main reasons for segmenting images is toconduct enhancement operations on the segmented portions. When the imageis segmented according to a binary segmentation method such as thatpreviously described, the subsequently applied enhancement operationsoften produce random variations in image enhancement, usually at theedges of the segmentation regions. Such random variations in movingsequences represent disturbing artifacts that are unacceptable toviewers. Image enhancement in the television setting includes bothglobal and local methods. While local enhancement methods are known,they are currently controlled by global parameters. For example, an edgeenhancement algorithm may adapt to the local edge characteristics, butthe parameters that govern the algorithm (i.e., filter frequencycharacteristics) are global—the enhancement operations that are appliedare the same for all regions of the image. The use of global parameterslimits the most effective enhancement that can be applied to any givenimage. Improved enhancement would be available if the algorithm could betrained to recognize the features depicted in different segments of theimage and could therefore allow the image enhancement algorithms andparameters that are optimum for each type of image feature to be chosendynamically.

However, one of the principle problems with the current state of the artis that it is essentially pixel-based. As the characteristics such ascolor and luminance within a segment may vary significantly from pixelto pixel, the determined segment probability function may includesignificant “noise-like” indicators. When the input video signal alsoincludes noise, the resultant segment probability function becomes evenmore noise-like. One method of reducing the noise-like indicators in theprobability distribution is to process it using a low-pass filter.However, such processing has the undesirable side-effect of removing thetexture in the segment of the image.

Hence, there a need for a method and system for reducing the effects ofthe noise in the determined segment probability function, whilemaintaining the image texture.

DETAILED DESCRIPTION OF THE INVENTION

As is known, video images may have significant areas or segments thatmay be identified as having substantially the same characteristics,e.g., color, luminosity, texture. For example, a segment of an image maycontain information related to a sky, i.e., blue color, smooth texture.Similarly, fields of grass may be identified by its green color andsemi-smooth texture. Such identification of areas, or segments of videoimages are more fully discussed in commonly assigned, co-pending relatedpatent application Ser. No. ______ and commonly assigned, co-pendingrelated patent application Ser. No. ______, which disclose determining aprobability function for each such segment identified.

FIG. 1 illustrates a pixel element view 100 of a portion of an imagesegment that is identified as having similar color, texture orluminosity. It will be understood that the principles of the presentinvention are applicable to each segmented determined in a video imageframe. In this exemplary illustration, pixel elements within anarbitrarily selected segment are organized into blocks of 8×8 pixelelements. It will be appreciated that while the present invention isdiscussed with regard to 8×8 pixel element blocks, the block size may beof any size or number of pixel elements, such as 7×7, 9×9, 16×16, etc.Conventionally, the block size is selected using a power of 2, i.e.,8×8, 16×16, 32×32, etc., as this allows transformation from one blocksize to another through simple binary shifts, i.e., dividing by powersof 2.

Furthermore, it would be understood that the block size need not besymmetrical as shown, but may contain any number of pixel elements ineither length or width. Only for the purposes of clearly illustratingand discussing the present invention, are the image pixel elements ofthe selected segment grouped into 8×8 blocks, represented as blocks110-180.

FIG. 2 illustrates a flow chart of an exemplary processing 200 inaccordance with the principles of the invention. In this exemplaryprocess 200, pixel elements are organized into blocks, such as thoseshown in FIG. 1, at block 210. At block 215, a probability functioncalculated for each pixel within a block is averaged or weighted usingknown averaging or weighting functions. At block 220, the average orweighted value of the probability function associated with each block isthen compared to a threshold value. When the average value of theprobability function of a block is greater than the threshold, a firstnew value is associated with the pixel block at block 225. However, whenthe average value of a block is less than the threshold value then asecond new value is associated with the pixel block at block 230. Forexample, a logical one may be associated with a block when its averageor weighted probability function value is greater than a threshold valueand a logical zero may be associated with a block when its average orweighted probability function value is less than a threshold value.Similarly, the first new value may be selected as a logical “0” and thecorresponding second new value may be selected as a logical “1”. In apreferred aspect of the invention, a threshold value may be establishedas a function of the video signal-to-noise ratio (SNR) within the block.Table 1 tabulates exemplary threshold and SNR values on a scale of 0 to255, wherein 255 is a maximum value. TABLE 1 SNR Threshold Value 20 dB67 26 dB 112 32 dB 130

FIG. 3 illustrates a flow chart an exemplary process 300 for improvingimage segmentation in accordance with the principles of the invention.In this exemplary process, a pixel block is selected at block 310. Atblock 320, an adjacent pixel block is selected at block 320. At block330, a next/subsequent pixel block is selected at block 330. At block340 a determination is made whether the value associated with theselected adjacent pixel blocks are substantially the same. If the answeris negative, then processing on the selected pixel block is completed.However, if the answer is in the affirmative, then a next/subsequentadjacent pixel block is selected at block 350. At block 360, adetermination is made whether each of the pixel blocks adjacent to theblock selected at block 310 have been processed. If the answer isnegative, then a determination is made whether the value of the selectednext/subsequent block is substantially the same as a previously selectedadjacent block at block 340. Processing continues as previouslydescribed.

However, if the answer at block 360 is in the affirmative, then adetermination is made at block 370 whether the value of the blockselected at block 310 is substantially similar to the value of theadjacent block selected at block 320. If the answer is in theaffirmative, then processing on the block selected at block 310 iscompleted. However, if the answer is negative, then the value of theblock selected at block 310 is altered to correspond to the value of theadjacent block selected at block 320. Accordingly, the anomaly valueassociated with the selected is removed and made comparable to thevalues of the adjacent blocks.

For example, a block associated with a logical zero value may have allof its associated adjacent pixel blocks having an opposite value oflogical one. In this, case, the block associated with the anomalouslogical zero value is “removed” by setting its associated value to alogical one value, similar to all the adjacent block associated value.Similarly, if a block with an isolated logical one value is surroundedby blocks associated with a logical zero value, the anomalous logicalone value is removed by setting the value to a logic zero.

Returning to FIG. 1, for example, the value associated with block 130may be altered when the value associated with each of blocks 110, 115,120, 135, 125, 140, 145, and 150 are substantially the same anddifferent than the value associated with block 130.

In one aspect of the invention, the value associated with each block maythen be used to control the processing that is to be done for each pixelwithin the block. For example, one form of pixel-level processing thatmay be performed is determine whether a noise filter must be turned onduring the processing of each pixel in the block. This method isadvantageous to strike a balance between reduced image noise andmaintaining appropriate textual information. In another aspect, thevalues associated with each block may be used to control forms ofprocessing such as modifying the edge sharpness or color of a regiondifferently than other regions.

FIG. 4 illustrates an exemplary embodiment of a system 400 that may beused for implementing the principles of the present invention. System400 may represent a television transmitting or receiving system,desktop, laptop or palmtop computer, a personal digital assistant (PDA),a video/image storage apparatus such as a video cassette recorder (VCR),a digital video recorder (DVR), a TiVO apparatus, etc., as well asportions or combinations of these and other devices. System 400 maycontain one or mores sources 410 which are in communication withprocessor system 401 via one or more networks 420. Processor system 401is then further in communication with one or more TV displays 450 orMonitors 460 via network 440. Processor system 401 may contain one ormore input/output devices 402, processors 403 and memories 404, whichmay access one or more sources 410 that contain video images. Sources410 may be stored in permanent or semi-permanent media such as atelevision transmitter or receiver, a VCR, RAM, ROM, hard disk drive,optical disk drive or other video image storage devices, real timedisplay containing analog or digital images. Sources 410 mayalternatively be accessed over one or more network 420 connections forreceiving video from a server or servers over, for example a globalcomputer communications network such as the Internet, a wide areanetwork, a metropolitan area network, a local area network, aterrestrial broadcast system, a cable network, a satellite network, awireless network, or a telephone network, as well as portions orcombinations of these and other types of networks.

Input/output devices 402, processors 403 and memories 404 maycommunicate over a communication medium 406. Communication medium 406may represent, for example, a bus, a communication network, one or moreinternal connections of a circuit, circuit card or other apparatus, aswell as portions and combinations of these and other communicationmedia. Input data from the sources 410 is processed in accordance withone or more software programs that may be stored in memories 404 andexecuted by processors 403. Processors 403 may be any means, such asgeneral purpose or special purpose computing system, or may be ahardware configuration, such as a laptop computer, desktop computer,handheld computer, dedicated logic circuit, integrated circuit,Programmable Array Logic (PAL), Application Specific Integrated Circuit(ASIC), etc., that provides a known output in response to known inputs.

In one embodiment, the coding and decoding employing the principles ofthe present invention may be implemented by computer readable codeexecuted by processor 403. The code may be stored in the memory 404 orread/downloaded from a memory medium such as a CD-ROM or floppy disk(not shown). In another and preferred embodiment, hardware circuitry maybe used in place of, or in combination with, software instructions toimplement the invention. For example, the elements illustrated hereinmay also be implemented as discrete hardware elements or as programmabledevices operable to execute coed.

After processing the input data, processor 403 may cause the processeddata to be transmitted to television display 480 or monitor 490 vianetwork 470. As will be appreciated, networks 420 and 440 may be aninternal network among the components, e.g., ISA bus, microchannel bus,PCMCIA bus, etc., or an external network, such as a Local Area Network,Wide Area Network, POTS network, or the Internet.

In one aspect of the invention, the term computer or computer system mayrepresent one or more processing units in communication with one or morememory units and other devices, e.g., peripherals, connectedelectronically to and communicating with the at least one processingunit. Furthermore, the devices may be electronically connected to theone or more processing units via internal busses, e.g., ISA bus,microchannel bus, PCI bus, PCMCIA bus, etc., or one or more internalconnections of a circuit, circuit card or other device, as well asportions and combinations of these and other communication media or anexternal network, e.g., the Internet and Intranet.

1. A method for improving the quality of a video image (100) into aplurality of blocks (110, 115, 120) comprising the steps of: associatinga value to each of said blocks; and altering said associated valuecorresponding to a selected one of said blocks when each of saidassociated values of blocks adjacent to said selected block is differentthan said selected block associated value.
 2. The method as recited inclaim 1, wherein said block associated value is a first value (225) whensaid block probability distribution is greater than a selectedthreshold, otherwise said block value is a second value (230).
 3. Themethod as recited in claim 2, wherein said block probabilitydistribution (215) is representative of an average of a probabilitydistribution associated with each pixel in said block.
 4. The method asrecited in claim 2 wherein said threshold is selected as a percentage ofsaid block probability distribution.
 5. The method as recited in claim 2wherein said threshold in relation to a signal-to-noise ratio in saidblock.
 6. A system for improving the quality of a video image (100)segmented into a plurality of blocks (110, 115, 120) of known sizecomprising: means for associating a value to each of said blocks; andmeans for altering said associated value corresponding to a selected oneof said blocks when each of said associated values of blocks adjacent tosaid selected block is different than said selected block associatedvalue.
 7. The system as recited in claim 6, wherein said blockassociated value is a first value (225) when said block probabilitydistribution is greater than a selected threshold, otherwise said valueis a second value (230).
 8. The system as recited in claim 7, whereinsaid block probability distribution is representative of an average of aprobability distribution associated with each pixel in said block. 9.The system as recited in claim 7, wherein said threshold is selected asa percentage of said block probability distribution.
 10. The system asrecited in claim 9, wherein said threshold is selected in relation to asignal-to-noise ratio within said block.